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ABSTRACT 



A weighted factors promotion system was field tested 
and validated using data from the FY 69-B promotion cycle of the 
Alaskan Air command. The final sample included 2,290 
promotion-eligible airmen in grades E-3 through E-6. The weighted 
factors composite score, excluding a promotion board component, gave 
airmen the same relative rank within selected Air Force Specialties 
as did the promotion board evaluations under the present operational 
system, overlaps between the two ranks imply promotion of the same 
individuals by both systems. However, inconsistencies and 
unexplainable discrepancies in the ranking by promotion board scores 
were found in some few instances. It was concluded that, within the 
specialties analyzed, practically all the individuals promoted by the 
board system would also have been promoted under the weighted factors 
system. If it can be assumed that the sample was representative of 
the Air Force-wide population of promotion-eligible airmen in grades 
E-3 through E-6, then it can be further assumed that the weighted 
factors system provides a valid airman promotion system in which the 
selection criteria are visible and equitable. (Author) 
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ABSTRACT 



A weighted factors promotion system was field-tested and validated using data from 
the FY 69-B promotion cycle of the Alaskan Air Command. The final sample included 
2,290 promotion-eligible airmen in grades E-3 through E-6. The weighted factors 
composite score excluding a promotion board component gave airmen the same relative 
rank within selected Air Force Specialties as did the promotion board evaluations under 
the present operational system. Overlaps between the two ranks imply promotion of the 
same individuals by both systems. However, inconsistencies and unexplainable 
discrepancies in the ranking by promotion board scores were found in some few 
instances. It was concluded that, within the specialties analyzed, practically all the 
individuals promoted by the board system would also have been promoted under the 
weighted factors system. If it can be assumed that the sample was representative of the 
Air Force-wide population of promotion-eligible airmen in grades E-3 through E-6, then it 
can be further assumed that the weighted factors system provides a valid airman 
promotion system in which the selection criteria are visible and equitable. 



SUMMARY 



Koplyay, J.B. Field test of the weighted airman promotion system: Phase IL Validation of the system for 
grades EA through E-7. AFHRL-TR-69-102. Lackland AFB, Tex.: Personnel Research Division, Air 
Force Human Resources Laboratory, May 1969. 

Problem 

In response to a research requirement levied by the Deputy Chief of Staff for Personnel, 
Headquarters, United States Air Force, the Personnel Research Division developed a model for a new 
airman promotion system. The. purpose of this new promotion system was to provide more visibility to 
each airman in terms of numeric values on selected variables indicating their relative performance on these 
variables. This, in turn, provides visible evidence and information about the reasons for non-promotion in 
terms of easily interpretable scores. 

The new promotion system (Weighted Airman Promotion System) was field-tested in the Alaskan Air 
Command (Koplyay, 1969). One of the major purposes of this field test was to obtain validation data. The 
operational definition of validation was assumed to be the relationship between the Weighted Airman 
Promotion System and the existing Promotion Board system in terms of the percentage of 
promotion-eligible individuals who would have been promoted by both systems. 

Approach 

To ascertain the predictive efficiency of the Weighted Airman Promotion System, a validity index was 
computed by identifying those individuals who would have been selected for promotion by both the 
Composite Score of the new system and the Board Score of the existing system. This validity index was 
computed for the entire sample of 2,290 airmen by grade (555 in grade E-3, 850 in grade E-4, 514 in grade 
E-5, and 371 in grade E-6), and was further analyzed by grade within selected specialties. 

Results 

The validity indices ranged from 43.5 percent (grade E-6) to 91.4 percent (grade E-3) for the entire 
sample. These results, however, are lower-limit estimates for the validity of the new system, since the 
existing promotion system operates on the basis of differential quotas assigned to the various specialties. 
The validity indices within grade and selected specialties ranged from 84 percent to 100 percent with 
two-thirds of the specialties analyzed having validity indices of 100 percent. 

Conclusions 

The Weighted Airman Promotion System gave most airmen the same relative ranking within their 
specialty as did the Promotion Board system. In other words, with a limited number of exceptions, the 
same individuals would have been promoted using either system. 

If the sample in the study can be considered to be representative of the Air Force-wide population of 
airmen in grades E-3 through E-6, it can be further assumed that the Weighted Airman Promotion System is 
not only a visible system but it is valid in the sense of the operational definition of validity used in this 
study. 

This summary was prepared by J.B. Koplyay, Statistical Analysis Branch, Personnel Research 
Division, Air Force Human Resources Laboratory. 
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FIELD TEST OF THE WEIGHTED AIRMAN PROMOTION SYSTEM: 
PHASE II. VALIDATION OF THE SYSTEM FOR 
GRADES E-4 THROUGH E-7 



I. INTRODUCTION 

The Weighted Airman Promotion System 
was developed, field-tested in the Alaskan Air 
Command, and evaluated in response to a research 
requirement levied by the Deputy Chief of Staff 
for Personnel, Headquarters, United States Air 
Force. Briefly, the system provides for selection of 
airmen for promotion to grades E-4 through E-7 
on the basis of seven weighted factors designed to 
insure visibility and equitable application of the 
selection criteria. One of the seven factors is a 
promotion board evaluation. The extent to which 
inclusion of this board score as a component in the 
proposed system changed the relative standing of 
promotion-eligible individuals was discussed in the 
report of the first phase of this study (Koplyay, 
1969). In general, the initial findings indicated 
that inclusion of the board component made only 
an insignificant difference in the ranking of airmen 
who were eligible for promotion. The seven factors 
and their computations or weights are shown in 
the appendix. 

One of the major purposes of the Alaskan 
Air Command field test of the proposed weighted 
factors promotion system was to obtain validation 
data. Validation is a very general concept, and it 
can be interpreted in a variety of ways. 

In this report, the operational definition of 
validation is assumed to be the relationship 
between the weighted factors composite score, 
excluding the average board score, and the actual 
promotion actions. Since promotions under the 
existing system are based on promotion board 
scores, validation of the weighted factors system 
dealt with analyses of relationships between the 
weighted factors system and the promotion board 
system. That is, if the same individuals were 
identified for promotion by the weighted factors 
system as were actually promoted under the 
existing system, then the weighted factors 
composite score can be considered a valid 
indicator (predictor) of promotion. In this 
context, it is always true that any unreliability in 
the criterion variable places an upper limit on the 
validity of any predictor. If the promotion board 
score has a less than perfect relationship with 
promotion actions, it is impossible to develop any 



kind of system to duplicate it. In this report, two 
general indicators are used to express the validity 
of the proposed weighted factors system. One is 
the comparison of the weighted factors composite 
scores with actual promotion actions. The other 
reflects directly the relationship between the 
proposed system and the promotion board system. 

This report focuses primarily on three 
specific areas: 

1 . The relationship between promotion 
board score and promotion action. 

2. The agreement between rank standing on 
the weighted factors composite score excluding 
the board score and rank standing on the 
promotion board score for the total sample and 
for selected Control Air Force Specialties. 

3. The analysis of groups with large 
discrepancies between rank on the weighted 
factors composite score and rank on the board 
score. 



II. PROCEDURE 

Description of the Sample 

The Alaskan Air Command provided data 
for 2,835 airmen in grades E-3 through E-6 who 
were eligible for promotion in the FY 69-B cycle. 
All factors except the Promotion Fitness 
Examination Scores were provided on punched 
cards by Consolidated Base Personnel Offices at 
Elmendorf and Eielson Air Force Bases. The 
Promotion Fitness Examination was administered 
separately, and scores were added to the record of 
each airman in the sample. Promotion board scores 
were given in raw score form as the sum of the 
ratings of individual promotion board members. 
There were both three-man and five-man boards. 

Treatment of the Data 

A number of steps were taken to obtain 
usable and meaningful data for analysis. 

1. Promotion Fitness Examination scores 
were added to the data tape containing the card 
images of punched cards provided by the Alaskan 
Air Command. 
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2. Records were excluded from further 
analyses if Specialty Knowledge Test scores or 
Promotion Fitness Examination scores were 
missing, or if non-matching and uncorrectable 
serial numbers and names were detected during the 
merging of input data tapes. As a result, 2,290 
airmen (555 in grade E-3, 850 in grade E-4, 514 in 
grade E-5, and 371 in grade E-6) were retained in 
the study for further analyses. 

3. Promotion board raw scores were 
converted to a scale, with a 100-point maximum 
score (see Appendix). Each raw score was divided 
by the appropriate number of board members, and 
the quotient '„as multiplied by 10. The resulting 
score gave an average board score for each airman. 
(Before adopting the procedure of generating the 
average board score, a preliminary analysis of the 
data had shown that the variability of the resulting 
scores would be approximately constant among 
grades.) In case of ties in average, board scores, an 
attempt was made to break these ties by 
consideration of the Time-in-Grade, Airman 
Performance Report, and Decorations scores, in 
that order. (This was in accordance with the policy 
of the promotion board, but it did not eliminate 
ties altogether.) The average board score, then, was 
added to each airman’s record. This score is 
referred to simply as the Board Score in the 
remainder of this report. 

4. Time-in-Grade and Time-in-Service scores 
were recomputed using Date of Rank, Total Active 
Federal Military Service Date, and the date 1 
January 1969 for airmen in grade E-3 and 1 May 
1969 for airmen in grades E-4 through E-6. The 
recomputed values were distributed against those 
supplied by the Alaskan Air Command. With the 
exception of one case, which was correctable, the 
recomputed values agreed with the original values. 

5. Total weighted factors composite scores 
were computed by adding the relevant factors (see 
Appendix) in two ways: (a) by excluding the 
average board score and ( b ) by including the 
average board score. Results of the Phase I analysis 
had indicated conclusively that inclusion of a 
board component in the weighted factors 
composite would introduce only trivial changes in 
a comparable composite which excluded a board 
component. Thus, the weighted factors composite 
scores including the board component were 
computed and recorded, but were not used in any 
of the analyses. The weighted factors composite 
score excluding the board component is referred 
to as the Composite Score in the remainder of this 
report. 



6. Within each grade, airmen were ranked 
on both the Composite Score and the Board Score. 
These created ranks were added to each record on 
the data tape. In general, the higher the score on a 
particular variable, the lower the corresponding 
rank value, For example, a rank value of 1 was 
assigned to the individual who had the highest 
score. Thus, high Composite Scores and high 
Board Scores are reflected in low rank values, 

7. During the analysis, a list was obtained of 
the promotion-eligible airmen who were actually 
promoted. This information was added to the data 
tape as an additional score of i if the airman was 
promoted and 0 if he was not. The variable thus 
created served as the indicator of promotion 
action. 

8. Data available for the weighted factors 
promotion system were distributed by single 
factors and by combinations of several factors. 
From the distribution by Control Air Force 
Specialty Code, groups with sufficient frequencies 
were selected as subgroups for more detailed 
analysis. 

9. Since one of the major factors in actual 
promotion is the quota allocated to a particular 
specially, grouping of cases by specialties within 
grade seemed to be essential for meaningful 
analysis. Thus,, airmen were re-ranked within their 
particular specialties and grade on both the 
Composite Score and the Board Score. These new 
ranks were used only when the groups were 
analyzed within specialties, 

III. ANALYSES 

Relationship Between Board Score 
and Promotion Action 

Traditionally, promotions have been made 
primarily on the basis of promotion board scores; 
however, if the promotion quota is either very 
high or very low, the board score is less of a 
determining factor. When the quota is high, for 
example, eligible airmen within a wide range of 
board scores from high to low are selected. If the 
quota is low, eligible airmen with both high and 
low board scores are not selected, while only those 
with the very highest scores are selected. When 
many eligible airmen receive tied board scores, but 
the quota does not permit all such airmen to be 
promoted, a promotion/non-promotion decision 
must be made. Promotion boards have generally 
accomplished this by taking a “second look” at 
the selection folders of the airmen with tied board 



9 ; 

2 



scores to consider other criteria in making the se- 
lections. For these reasons, it was anticipated that 
the relationship might be less than perfect between 
average board scores and promotion/non- 
promotion actions. To obtain empirical evidence 
on this question, the relationship between the 
Board Score and the actual promotion action was 
determined within each of several specialties and 
for the four grade levels involved. 

The most widely used technique to ascertain 
relationships is computation of a coefficient of 
correlation. Correlation coefficients, obtained as a 
by-product of the regression analyses performed 
and reported separately under Phase I of this study 
(Koplyay, 1969), are shown in Table 1. 
Interpreting correlation coefficients, particularly 
when the correlation is between a continuous 
variable (such as Board Score) and a binary 
variable (such as promotion/non-promotion), is 
extremely difficult because the correlation not 
only depends on the strength of the relationship 
between the two variables, but it is also affected 
by other factors such as the “split” on the binary 
variable (e.g., the proportion of promoted airmen 
to non-promoted airmen) and the number of 
unique values on the non-dichotomized variable 
(e.g., the number of ties in Board Score), For 
example, in the case of grade E-3, specialty 702, it 
was known that 24 of the 25 eligible airmen were 
actually promoted. With the exception of one 
airman in this grade and specialty, all eligible 
airmen were promoted regardless of their Board 
Score. Thus, the computed correlation indicates a 
weak relationship between Board Score and 
promotion action. The correlation coefficients 
indicated in Table 1, therefore, should be 
interpreted with caution and in light of any 
circumstances which might yield spurious results. 



From Table 1, it is apparent that the Board 
Scores correlated less than perfectly with the 
actual promotions within grade. Correlations 
within grade by selected specialties ranged from 
.1989 to .8029, showing both relatively low and 
relatively high correlations. As already suggested, 
however, these correlations reflect a somewhat 
blurred picture of the precise relationship between 
the Board Score and actual promotion. The low 
values could be a result of any one or all of several 
reasons: (a) extreme split between promoted and 
non-promoted subgroups, ( b ) tied Board Scores 
among those competing for promotions, and (c) 
inversions in which eligible airmen with lower 
Board Sco/cs were promoted over airmen with 
higher Board Scores. Further analysis of the data 
indicated that there were only two inversions: 
hence, this reason can be eliminated as a factor to 
account for the observed low correlations. There 
were, however, occasional extreme splits and a 
high incidence of ties in Board Score (as discussed 
in Phase l of this study)* and apparently these r.rc 
the major factors accounting for the tow 
correlations which do exist, 

There is one other possible explanation for 
the less than perfect relationship between Board 
Scores and the actual promotion actions. The 
results of Phase { of the study had revealed 
statistically significant differences between the 
means of the Board Scores for the two bases, 
Elmendorf and Eielson, for airmen in grade E-3. 
Since these differences only existed for grade E-3, 
and the promotion/non-promotion split for this 
grade was consistently extreme across specialties, 
it was felt that possible effects of differences in 
Board Score means (which generally were 
numerically small although statistically significant) 
could not have been completely separated from 



Table i. Correlations Between Board Score and Promotion Action Within Each Pay Grade 
for Total Sample and for Selected Specialties 



Corr elation by SpsdaSty Subdivision 

Pay Correlation 



Grade 


by Pay Grade 


431 


631 


645 


647 


702 


an 


E-3 


.4709 


.5145 


.4538 


.6563 


.6450 


.1989 


.5273 


E-4 


.5620 


.8029 


.5638 


.7124 


.5590 


.7259 


.5426 


E-5 


.5609 


.7101 


.3365 


.5469 


.7324 


.7016 


.7199 


E-6 


.5194 


.5675 


.6283 


.7495 


.7255 


.4879 


.3864 





the adverse effect of the extreme splits on the 
correlation between Board Score and actual 
promotion action. Furthermore, subsequent 
analyses indicated that, with the exception of two 
inversions (i.e., cases in which airmen with lower 
Board Scores were promoted over airmen with 
higher Board Scores), all airmen who were 
promoted, regardless of their base of assignment, 
had Board Scores as high as or higher than those 
who were not promoted. This finding seemed to 
justify the conclusion that extreme promo- 
tion/non-promotion splits are mainly responsible 
for the less than perfect correlation between Board 
Score and promotion action as shown in this 
study. 

Agreement Between Promotion 
Predictions and Actual Promotion 
Outcomes Within Pay Grades 

Another phase of the analysis dealt with the 
validation of the Composite Score against actual 
selection, and then comparison of that relationship 
with the relationship between Board Score and 
promotion action. To appraise the validity of the 
proposed weighted factors composite score, it was 
assumed that the system had, in fact, been used as 
the basis for determining promotion/non- 
promotion decisions. The resulting outcomes 
derived from this hypothetical application of the 
system would then be considered “predictions” of 
the selection/non-selection outcomes as indicated 
by the actual promotion actions. Thus, if an 
eligible airman would have been promoted by the 
proposed system and was, in fact, promoted, then 
the proposed system provided a correct prediction 
of the promotion outcome. The validity of the 
proposed system is operationally defined for the 
purpose of this analysis in terms of the number of 
correct promotion predictions and the proportion 
of correct promotion predictions. Ranks on the 



Composite Score were used as the basis for 
selecting individuals to receive hypothetical 
promotions under the proposed system. 

To ascertain the predictive efficiency of the 
proposed system, a validity index was computed 
by identifying those individuals who would have 
been selected for promotion by both the 
Composite Score and the Board Score. The ratio 
of the number of such individuals to the actual 
number of promotions is operationally defined as 
promotion overlap. Table 2 summarizes the 
correct promotion predictions, or promotion 
overlap, for the Composite Score by pay grade. 

An example drawn from the data in Table 2 
will illustrate the predictive validation. For grade 
E-5, the number of correct promotion predictions 
of the proposed system was determined by 
distributing and ranking the Composite Scores of 
the 514 E-5 airmen. With a promotion quota of 
89, the 89 airmen with the highest scores would 
presumably have been selected for promotion and 
the remaining 425 non-selected. Of the 514 airmen 
in grade E-5, 56 airmen were selected for 
promotion by both the Composite Score and the 
Board Score. This represents a promotion overlap 
of 62.9 percent (or 56/89). In other words, 62.9 
percent of those airmen whose rank on the Board 
Score was better than or equal to the promotion 
quota of 89 were also ranked among the top 89 on 
the Composite Score. Although this does not mean 
that these 56 airmen received the same rank on 
both Composite Score and Promotion Score, the 
end result of promotion selection would be the 
same. That is, one airman could have been ranked 
1 on the Composite Score and 89 on the Board 
Score; nevertheless, he would have been selected 
for promotion by either system since the quota 
was 89. 



Table 2. Correct Promotion Predictions by Composite Score 
Within Each Pay Grade 











Promotion Overlap 








Correct 


Proportion of 










Predictions 


Correct Predic- 


Percentage 


Pay 


Number 


Number 


by Composite 


tions to Pro- 


of 


Grade 


Eligible 


Promoted 


Score 


motions 


Overlap 


E-3 


555 


478 


437 


437/478 


91.4 


E-4 


850 


212 


141 


141/212 


66.5 


E-5 


514 


89 


56 


56/89 


63.9 


E-6 


317 


46 


20 


20/46 


43.5 
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If there were one single promotion quota for 
each pay grade, rather than different quotas for 
different specialties as is the case, the data in Table 
2 would adequately answer the question of valid- 
ity of the proposed system. In general, the results 
in Table 2 indicate only a moderate level of valid- 
ity of the system for actual promotion/non- 
promotion outcomes, with the percentage of 
correct promotion predictions ranging from 43.5 
percent to 91.4 percent. Furthermore, this overlap 
in promotion predictions is less than 70 percent 
for three of the four grades in question. These 
results, however, should be considered as lower- 
limit estimates for the validity of the proposed 
system and should be interpreted with caution 
since the existing promotion system does operate 
on the basis of differential quotas. All the airmen 
within a pay grade are ranked on promotion board 
score, and then the quota for each specialty is 
filled going down the list from high score to low. 
Under this system, it is quite possible that airmen 
with high promotion board scores will not be 
promoted and airmen with lower board scores will 
be promoted. 

To clarify this, let us assume that a 
particular airman ranks 10 on promotion board 
score, and five of his fellow airmen in the same 
specialty rank 9 or better on the board score. 
Suppose, further, that this particular specialty has 
a quota of four promotions. The airman in 
question will not be promoted since there are five 
persons in his specialty who rank better than he 
does on the board score. Now, let us assume that 
another airman, who is in the same pay grade as 
the first airman but in a different specialty, ranks 
55 on the board score* Assuming, further, that this 
second airman’s specialty has a quota of five 
promotions and that no other airman in his group 
has a rank better than 55 on the board score, this 
airman will be promoted since his board score rank 
is the highest within his specialty. The promotion 
outcome is favorable for the second airman and 
unfavorable for the first airman in spite of the fact 
that he had a better promotion board score in 
terms of the comparison across pay grade. 

Agreement Between Promotion Predictions 
and Actual Promotion Outcomes 
Within Selected Specialties 

Because of the differences by specialty, a 
further step was undertaken to analyze the 
relationship between the Composite Score and 
Board Score in terms of correct promotion 
predictions within particular specialties. Two new 
variables were created: rank on the Composite 



Score and rank on the Board Score within 
specialties. 

The distribution of airmen by Control Air 
Force Specialty Codes did not provide an adequate 
sample for all the specialties in the study. Most of 
the specialties did not have enough individuals 
within each pay grade to permit a meaningful 
analysis. It was possible, however, to select six 
specialties with a large enough sample within each 
pay grade for further analysis. The same 
procedures as used in the comparison for the total 
sample were used for the within-specialty analyses. 
The only exception was that each airman was 
re-ranked on the Composite Score and the Board 
Score within his specialty. 

Table 3 summarizes the promotion 
predictions within grade and by selected 
specialties. These results clearly imply that there 
was a very high degree of agreement in promotion 
prediction between the Composite Score and the 
Board Score. The lowest promotion overlap was 
83.33 percent, and in 16 out of the 24 specialty- 
by -grade combinations, the agreement was 100 
percent. One must conclude that the Composite 
Score would have promoted the same individuals 
in the 16 specialties analyzed as did the Board 
Score. Within the remaining 8 specialties, the 
overlap of predicted promotions ranged from 83.33 
percent to 96.84 percent. 

Analysis of the specialties in which 
agreement fell short of 100 percent indicates that 
the policy used in breaking tied promotion board 
scores was not applied with complete consistency. 
This may account for a small number of cases 
which were predicted for promotion by the Board 
Score but not by the Composite Score, and vice 
versa. Also, it appears that in some few cases the 
board rated an individual high if his Airman 
Performance Report variable was high, regardless 
of low scores on other variables. In short, the same 
factors which contribute to the lack of complete 
visibility of the selection criteria under the 
promotion board system probably also account for 
the less than perfect relationship between the 
Composite Score and the Board Score. 

Sample Cases of Inconsistencies Between 
Composite Scores and Board Scores 

To illustrate the occasional inconsistencies 
of the promotion board scores in light of the 
components of the weighted factors system, the 
scores and ranks of four airmen in grade E-4 are 
compared in Table 4. It is apparent from the table 
that Airman A’s performance was clearly much 



Table 3. Correct Promotion Predictions by Composite Score 
for Selected Specialties Within Each Pay Grade 



Spec- 

ialty 


Number 

Eligible 


Number 

Promoted 


Correct 
Predictions 
by Composite 
Score 


Promotion 

Proportion of 
Correct Predic- 
tions to Pro- 
motions 


Overlap 

Percentage 

of 

Overlap 


431 


49 


46 


Pay Grade E-3 
44 


44/46 


95.6 


631 


23 


22 


22 


22/22 


100.0 


645 


41 


37 


37 


37/37 


100.0 


647 


32 


29 


28 


28/29 


96.6 


702 


25 


24 


24 


24/24 


100.0 


811 


108 


95 


92 


92/95 


96.8 


431 


55 


28 


Pay Grade E-4 
25 


25/28 


89.3 


631 


74 


6 


5 


5/6 


83.3 


645 


57 


9 


8 


8/9 


88.9 


647 


35 


4 


4 


4/4 


100.0 


702 


50 


11 


10 


10/11 


90.9 


811 


35 


4 


4 


4/4 


100.0 


431 


34 


13 


Pay Grade E-5 
13 


13/13 


100.0 


631 


42 


1 


1 


1/1 


100.0 


645 


44 


10 


10 


10/10 


100.0 


647 


10 


2 


2 


2/2 


100.0 


702 


42 


6 


5 


5/6 


83.3 


811 


25 


3 


3 


3/3 


100.0 


431 


31 


2 


Pay Grade E-6 
2 


2/2 


100.0 


631 


17 


2 


2 


2/2 


100.0 


645 


25 


5 


5 


5/5 


100.0 


647 


11 


3 


3 


3/3 


100.0 


702 


33 


3 


3 


3/3 


100.0 


811 


13 


2 


2 


2/2 


100.0 





'$r 




Table 4. Comparison of Composite Score and Board Score Variables 
for Four Sample Cases 



Variable 




Sample Cases 




Airman A 


Airman B 


Airman C 


Airman D 


Specialty Knowledge Test Score 


80 


30 


95 


60 


Promotion Fitness Examination Score 


85 


15 


95 


75 


Decorations Score 


0 


0 


0 


0 


Airman Performance Report Score 


125 


128 


135 


135 


Time-in-Grade Score 


44 


10 


12 


7 


lime-in-Service Score 


23 


7 


8 


6 


Composite Score Rank 


2 


792 


8 


256 


Board Score Rank 


759 


77 


230 


2 


Promotion Action 


0 


1 


1 


1 



superior to Airman B’s. In fact, Airman B scored 
very poorly in general. Yet, Airman A ranked 759 
on the Board Score and was not promoted. 
Airman B, on the other hand, ranked 77 on the 
Board Score (792 on the Composite Score) and 
was promoted. 

Airman C obtained scores far superior to 
those of Airman D. Although both were 
promoted, C ranked 230 on the Board Score (with 
superior performance on the Composite Score); D 
ranked 2 on the Board Score (with poor 
performance on the Composite Score). 

The inconsistencies which are present in the 
sample indicate that the board must be evaluating 
some “invisible” additional factors besides those 
included in the weighted factors composite in 
these cases. Since rankings on the Board Score in 
the total sample within grade (across specialties) 
are not influenced by quotas assigned to the 
specialties, inferences about inconsistencies in 
Board Score rankings are valid. Although it is clear 
in these cases that the Board Score reflects a 
weighting of information from the selection folder 
which is not included in the Composite Score, it 
should be reemphasized that the actual promotion 
outcome is a function not only of the Board Score 
but also of the quota assigned to a specialty. Thus, 
while a low Board Score decreases the likelihood 
of promotion, and a high Board Score increases 
the likelihood of promotion, it does not 
necessarily follow that a high rank-value on the 
Board Score results in non-promotion and a low 
rank-value on the Board Score guarantees 



promotion. That is, a rank (within grade) of 230 
(Airman C in Table 4) could have represented a 
relative rank (within specialty) good enough to be 
promoted based on the quota for that specialty. 

Analysis of Rank Discrepancies 

The sample within each grade was divided 
into four mutually exclusive groups based on ranks 
on the Composite Score and the Board Score: 

Group 1. Individuals whose ranks on both 
scores were less than or equal to the 
promotion quota. 

Group 2. Individuals whose rank on the 
Composite Score was less than or equal to 
the promotion quota, and whose rank on the 
Board Score was greater than the promotion 
quota. 

Group 3. Individuals whose rank on the 
Composite Score was greater than the 
promotion quota, and whose rank on the 
Board Score was less than or equal to the 
promotion quota. 

Group 4. Individuals whose ranks on both 
scores were greater than the promotion 
quota. 

Groups 1 and 4 are the “agreement” groups; 
groups 2 and 3 are the “discrepancy” groups. ITie 
latter groups, 2 and 3, were analyzed on the 
components of the weighted factors composite 
score. The results of these analyses are summarized 
in Table 5. 
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Comparison of the groups in terms of the six 
components of the weighted factors composite 
score reveals the following findings. 

1. Specialty Knowledge Test Scores. Group 
2 (high Composite Scores but low Board Scores) 
was superior to group 3 (low Composite Scores 
but high Board Scores) in all grades. The means 
differed by more than 20 points except in grade 
E-5 where the difference was 13.9 points. In all 
cases, the differences were statistically significant. 

2. Promotion Fitness Examination Scores. 
The means of group 2 were superior to the means 
of group 3 in all grades. The differences ranged 
from 23.6 points (E-6) to 44.5 points (E-3). 

3. Decorations Scores. There were no 
statistically significant differences between the 
means of group 2 and group 3. 

4. Airman Performance Report Scores. 
There was a statistically significant difference 
between means of group 2 and group 3 in favor of 
group 3 in all grades. These differences, however, 
were relatively small numerically (18.7 for E-3, 5.0 
for E-4, 6.2 for E-5, and 2.9 for E-6). 

5. Time-in-Grade Scores. With the exception 
of grade E-3, means of group 2 and group 3 
differed significantly in favor of group 2 in all 
grades. 

6. Time-in-Service Scores. Means of group 2 
and group 3 differed significantly in favor of group 
2 in all grades except grade E-3. The difference 
was relatively small but statistically significant. 

With the exception of the Decorations 
factor, where the two discrepancy groups were 
comparable, and the Airman Performance Report 
factor, where a small but significant difference in 
favor of group 3 was found, group 2 was superior 
on all factors. 



IV. SUMMARY 

The weighted factors promotion system was 
field-tested with the cooperation of the Alaskan 
Air Command. From among' 2,835 
promotion-eligible airmen at Elmendorf and 
Eielson Air Force Bases, a final sample of 2,290 
airmen in grades E-3 through E-6 was obtained. 
Data were analyzed to determine the relationship 
between the promotion board score (Board Score) 
and the weighted factors composite score 
excluding the board score (Composite Score). In 
addition, actual promotion actions were related to 
promotion board scores. Overlaps between rank on 
the Board Score and rank on the Composite Score 



were analyzed for the entire sample within each 
grade and for selected specialties. Discrepancies in 
ranks for a few individual cases were further 
analyzed. 

The results of the various analyses are 
summarized, and several conclusions are drawn 
regarding the relationship between the existing 
airman promotion system based on board 
evaluations and the weighted factors promotion 
system. 

1 . The relationship between the Board 
Score and actual promotion was less than perfect. 
Correlations ranged between .471 and .562 for the 
total sample. For selected specialties, the range 
was from .199 to .803. Although ties on the Board 
Scores may have been partially responsible for the 
observed low to moderate correlations, it is felt 
that the different promotion quotas for different 
specialties probably account for additional 
variability in the relationship between Board Score 
and actual promotion. 

2. Promotion overlap between Composite 
Score and Board Score ranged from 43 percent for 
grade E-6 to 91 percent for grade E-3 for the total 
sample. Further analysis within specialties resulted 
in almost complete overlap. When airmen were 
re-ranked on the Composite Score and on the 
Board Score within their specialties, the overlap 
was 100 percent in 16 out of the 24 groups 
analyzed. The overlap ranged between 84 percent 
and 96 percent in the other eight groups. This does 
not necessarily imply that the ranks on Board 
Score and Composite Score were the same. It does 
imply, however, that when re-ranked within a 
particular specialty, practically the same airmen 
would have been selected for promotion by both 
the Board Score alone and the Composite Score 
excluding the Board Score. Thus, it is apparent 
that the proposed weighted factors promotion 
system is highly valid in the sense that it provides 
results which are in close agreement with the 
operational system. 

3. Analyses of individual cases indicated 
occasional inconsistencies on the part of the 
promotion board in assigning Board Scores on the 
basis of the factors included in the weighted 
factors composite score. In some cases, airmen 
with superior scores on these factors were ranked 
lower by the board than others whose scores 
reflected poor performance. 

4. Most airmen either ranked high on both 
the Board Score and the Composite Score, or they 
ranked low on both. A relatively small number of 
cases ranked high on the Composite Score and low 
on the Board Score, while another small number 



ranked high on the Board Score and low on the 
Composite Score. Comparison of these 
rank-discrepancy groups revealed that, with the 
exception of the Airman Performance Report 
variable and the Decorations variable, the average 
score on each factor was higher for the group in 
which the rank discrepancy was in favor of the 
Composite Score than in the reverse discrepancy 
group. The Airman Performance Report mean was 
slightly higher in the group which ranked high on 
the Board Score and low on the Composite Score. 
However, the margin by which the Airman Per- 
formance Report factor was higher was relatively 
slight compared to the large differences on other 
factors revealed when the rank-discrepancy groups 
were analyzed. The Decorations factor was 
approximately equal in the two groups. 

To summarize the overall findings, then, the 
weighted factors composite score excluding the 
average board score gave most airmen the same 
relative ranking within their specialty as did the 
promotion board score. In other words, the same 
individuals would have been promoted using either 
system. On the other hand, when all promotion- 
eligible airmen within a pay grade were compared 
with each other, there were indications of 
inconsistencies in the promotion board’s judgment 
of the relative merits of performance in arriving at 
the promotion/non-promotion decision. There was 
a definite lack of visibility of selection factors in a 
few cases where airmen with superior Composite 



Scores were given low Board Scores. This does not 
imply that the promotion board had no valid 
reason for its judgment. It does suggest, however, 
that the reason would not be readily apparent to 
an airman under the current operational board 
process. 

Validation of the weighted factors 
promotion system was accomplished by ranking 
the individuals on their Composite Scores and 
comparing these ranks to actual promotion 
outcomes. It was concluded from the results that 
the weighted, factors promotion system would 
have promoted almost the same individuals within 
the specialties analyzed as were actually selected 
by the promotion boards. If the sample in the 
study can be considered to be representative of the 
Air Force-wide population of promotion-eligible 
airmen in grades E-3 through E-6, it can be further 
assumed that the weighted factors system provides 
a valid promotion system in which the selection 
criteria are visible. 
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APPENDIX: AIRMAN PROMOTION SELECTION FACTORS AND POINTS 1 



Grades E-4 through 2> 7 



Selection Factor 


Computation With 
Board Score 


Computation Without 
Board Score 


Maximum 

Points 


Percentage 


Maximum 

Points 


Percentage 


Specialty Knowledge Test (SKT) Score 


95 


17 


95 


21 


Promotion Fitness Examination (PFE) Score 


95 


17 


95 


21 


Time-In-Service (TIS) Score 


40 


7 


40 


9 


Time-In-Grade (TIG) Score 


60 


11 


60 


13 


Decoration Score 


25 


5 


25 


6 


Airman Performance Reports (APR) 


135 


25 


135 


30 


Board Evaluation 


100 


18 








550 


100 


450 


100 



Explanation of Factors 

The SKT and PFE will be administered annually. 

Points for the SKT and PFE scores will be actual percentile scores obtained in these tests (in 5-point 
increments), 

Time-in-Service will be computed by multiplying years of Total Active Federal Military Service by 2. Less 
than 6 months will count as 1 point; over 6 months will count a full year, 2 points. A cutoff score of 40 
points, for 20 years TAFMS, has been established, 

Time-in-Grade will be computed at the rate of Vi point per month up to a maximum of 120 months, 60 
points; 1 5 days or less will be dropped, 16 or more will count as a full month. 

Decorations will be assigned points according to their order of precedence. The maximum number of points 
attainable is 25. Decorations will count for promotion regardless of the military service in which they were 
earned. 

The Airman Performance Report score is obtained by multiplying the overall evaluation mean by 15. The 
mean is based on reports for a 5-year period prior to the eligibility date, not to exceed ten reports. 

The Board Score will be based on a review by the board that concentrates on those items not previously 
weighted; e.g,, education level and efforts to improve self in terms of formal education, technical 
knowledge, jetc. Reduced selection folder will consist of Category A favorable communications, APR word 
picture, and pages 2 and 4 of the Air Force Form 7. 



a From Kopiy ay, 1969, p. 11. 
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